fix(agc): fault image table entries the device or guest memory cannot hold - #6481

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

P2 source-review finding at e441ee7bc13c4f2136a8a349ce197abafe79fdcf: sampledImageFormatLimits caches nullopt when the query explicitly returns VK_ERROR_FORMAT_NOT_SUPPORTED (Texture.cpp:259–277). RequireSampledImageWithinDeviceLimits then falls back to broad VkPhysicalDeviceLimits (295–306). An otherwise valid, registered 8x8 entry can pass and reach cachedTexture / vkCreateImage with the same format, type, optimal tiling, transfer-destination/sampled usage and create flags. A missing query provider already throws; this nullopt represents an explicit rejection.

The Vulkan query contract defines that result as an unsupported combination. Broad dimension/layer limits cannot establish support for it; see also image-creation support requirements.

Please turn the explicit unsupported result into an ordinary descriptive entry refusal before creation. resolveImageTables can then record its per-entry fault through the existing runtime_error catch, while retaining VulkanError propagation for genuine driver failures.

Suggested regression in the existing recording mock: return VK_ERROR_FORMAT_NOT_SUPPORTED for a registered 8x8 entry, use a fresh physical-device identity to avoid a cached positive result, and assert that the query ran, the table element is rejected, and no vkCreateImage call was recorded. The present stub always returns success.

Source/specification review only. A native regression patch was prepared and checked for applicability, but remains uncompiled/unexecuted. No GPU failure or connection to the observed title device loss is established.

Confirmed at e441ee7: sampledImageFormatLimits cached nullopt for VK_ERROR_FORMAT_NOT_SUPPORTED, and RequireSampledImageWithinDeviceLimits then compared the entry with VkPhysicalDeviceLimits, so a registered 8x8 entry reached vkCreateImage with the combination the query had rejected.

Fixed in c578222: an unsupported combination is now refused before the entry's guest memory is read (the message names the VkFormat, image type and create flags), through the existing runtime_error path, so the entry faults when selected like the other refusals. The device-wide limits are no longer consulted. Any other query result still goes through Check and propagates as VulkanError.

Your regression is in tests/ImageTableEntryValidation.cpp as testUnsupportedImageCombination, unchanged apart from whitespace. Before the fix it fails with an 8x8 registered texture whose exact image combination is unsupported: the entry was served instead of faulting when selected; images created: 8x8x1/1 layers; with it, agc_driver_graphics and the 73 image-related AGC tests pass on the RTX 3060 (591.86, Windows Server 2025). The technical debt entry and the PR body's Tested section are updated.

AI-assisted: yes

ds-main5-c01: one image table entry's out-of-memory error stopped Demon's Souls (PPSA01342)

Windows Server 2025, RTX 3060 12 GB, driver 591.86, 32 GB RAM. The build was main a5e2a80c6 with an earlier head of this PR and other open PRs, among them #5987 and #6553, which print the two [gpu] texture: placement lines. Emulator log lines only:

stderr 11037   [gpu] texture 0x27ad00000 (format 22) has unreadable DCC keys at 0x909809240000; its texels are read as stored
stdout 457474  [src/Texture.cpp:348 Texture] Texture address=0x27ad00000 1024x1024 mips=1 layers=1793 dim=2 tile=4 format=22 vk=100 element=4
stderr 11044   [gpu] texture: device-local heap 0 uses 7970 of its 10053 MiB budget and no allowed host heap has room, 7520387072 bytes kept in device-local memory type 1; 1 such allocations, 7172 MiB
stderr 11045   [gpu] texture: device-local memory type 1 is out of device memory for 7520387072 bytes; placed in host memory type 0; 1 such allocations, 7172 MiB
stderr 11046   [videoout] presentation failed: AGC graphics: vkAllocateMemory buffer: allocation=7520387072 memoryType=4 reclaimed=1867581668 initialResult=-2 attempts=2: Vulkan result -2
stderr 11050   FATAL: abort() (SIGABRT) on thread 12272

• The entry (stdout L457474): a 1024x1024 format 22, tile 4 array of 1793 layers, 4 MiB x 1793 = 7,520,387,072 bytes. Its first four dwords match a render target the run uses with 3 and 4 layers. Its last slice (0x700) and its DCC address (0x909809240000, L11037) are not a T#. 1793 layers is under the device's 2048 and the surface starts in registered memory, so this PR's checks passed it.
• Memory (L11044–L11046): device-local heap 0 was at 7970 of its 10053 MiB budget. The image went to host memory type 0 (heap 1, 16334.9 MiB). Its 7,520,387,072-byte staging buffer in host type 4 (also heap 1) then failed with -2 (VK_ERROR_OUT_OF_DEVICE_MEMORY), after the buffer pool reclaimed 1,867,581,668 bytes.
• The stop (L11046–L11050): resolveImageTables rethrew the VulkanError, presentation failed, and the VideoOut thread called std::terminate (SIGABRT).
• Not system commit: private memory rose from 28.6 to 42.4 GB between 652 and 672 s, while commit stood at 70.8 of 95.9 GB. What ran out was Vulkan heap 1.

What changed on the branch (force-pushed, 46f639206 → ce99615b2):

• c72f290 refuses an entry whose level 0 is larger than the largest memory heap. This is an exact limit (VUID-vkAllocateMemory-pAllocateInfo-01713). The bound is the heap's size, not the budget: L11044 reports no host heap with budget room, yet L11045 shows the driver allocated the image there. This entry fits heap 1, so it is not refused by size.
• ce99615 turns an out-of-memory result from the entry's own image or upload buffers into a faulting entry, and resolves the table again on its next use. Every other VulkanError still stops the draw, including out-of-memory errors from a recorder submit or a storage image refresh. With it, this entry would stop the title only if a shader selected it.

This is one run, and it does not show whether a shader selects the entry.

AI-assisted: yes

Tested on Linux (Arch), c++ (GCC) 16.2.1 20260810, Release, BUILD_TESTING=ON, main at 5d9dcc42d2ec5401aebd089028b8d4d7410a77b4. GPU tests ran on lavapipe (Mesa 26.2.2, llvmpipe with LLVM 22.1.8).

This PR and #6402 break agc_driver_image_table together. main passes (2 of 2), each PR alone passes, and the pair fails (2 of 2):

overlapping guest allocation

Cause: this PR registers FloatTexels, UintTexels, SintTexels and DepthStorage once for the whole run in main() (ImageTable.cpp:1331). #6402 registers the same three texel arrays again inside RunStoreTests (ImageTable.cpp:996-1000 on its head), and the allocation registry rejects the overlap. Removing the registeredUint/Float/Sint registration in RunStoreTests makes the test pass on the merge.

Whichever PR lands second should drop its duplicate registration.

AI-assisted: yes

Thank you, @nicolasfalesy. Your reading is right. Our agc_driver_image_table change registered FloatTexels, UintTexels, SintTexels and DepthStorage once in main() for the whole run. #6402's RunStoreTests registers the same three texel arrays again, and the guest allocation registry throws "overlapping guest allocation" when an address is registered twice. Each PR passes alone, and the two fail together. We confirmed that from the code; we have not reproduced your lavapipe run.

What changed (new head d1f7857, was ce99615; rebased onto current main c842491):

• tests/execution/ImageTable.cpp: the four registrations are now one scoped helper (TexelRegistrations) that is alive only while the tests that sample those arrays run. RunStoreTests runs outside that scope, so nothing of ours is registered while feat(agc): store to images selected through image tables #6402 registers its own copies. No assertion of our tests changed.
• The rebase had one conflict, in docs/dev/debt/libSceAgcDriver.md. We kept main's new Pipeline journal entry and our extended AGC buffer allocations entry.

Checked on CPU, without a GPU:

• feat(agc): store to images selected through image tables #6402 (9161151) merged into this head with no conflicts, and agc_driver_image_table builds there.
• Rebasing our five commits one at a time onto main plus feat(agc): store to images selected through image tables #6402 has no conflicts either, and ends in the same tree as the merge, so the landing order should not matter.
• agc_driver_graphics (mock Vulkan, includes our ImageTableEntryValidation tests) passes on this head and on the merge with feat(agc): store to images selected through image tables #6402. tools/check_conventions.py --base upstream/main passes.

GPU, on the RTX 3060 (driver 591.86, Windows Server 2025): on this head agc_driver_graphics, agc_driver_bindless_images, agc_driver_image_sint8, agc_driver_image_table, agc_driver_image_table_cache, agc_driver_image_table_cache_bounds and agc_driver_image_table_reuse pass (7 of 7). On the merge with #6402, agc_driver_image_table passes (1 of 1).

The test change is folded into the first commit of this PR, so the push is a force-push; the difference to the previous head is tests/execution/ImageTable.cpp and the rebase conflict above.

AI-assisted: yes

@adriadam10 rebased to 84c1a4d9a and re-derived the allocation boundary in clean head 1be24da14e8a0e9afecfa4f558018ce19d1bd895.

The snapshot retains main's complete AllocateImageMemory reclaim/retry/host-fallback route inside ownTextureMemory. Only a final host/device allocation refusal becomes an entry fault. Its own image creation/binding, upload allocations and ordinary sRGB expansion are classified too; device loss, recorder submission and storage-copy expansion errors retain propagation. Main's multisampleViewType and scoped image-table registrations are preserved.

The historical 7.52 GB upload refusal was already in host heap 1 after reclaim; device-to-host fallback cannot provide more room there. I have qualified the body: that old run is not a test of this head or proof that the entry remains unselected. The format preflight's sRGB usage/expansion limits are also explicit.

The allocation mock now tracks the same image across retries. One temporary refusal must recover; persistent refusals fault the entry and permit the next one. Removing only the common allocator or only the ordinary sRGB classifier produces the intended new assertions. Both CPU/mock suites pass; the five required-Vulkan image-table adjacency cases pass on RTX 3060 in 5.22 s without skips. Conventions and a clean merge-tree against 676d42396 pass separately; tests ran on the exact candidate. No new title or Linux result is claimed.

GPUs tested: NVIDIA GeForce RTX 3060 12 GB, driver 591.86 (32.0.15.9186), Vulkan 1.4.325, Windows Server 2025 10.0.26100.

AI-assisted: yes (Claude Code and Codex)

Co-Authored-By: Codex noreply@openai.com

One coverage distinction for coordination: RequireSampledImageWithinDeviceLimits is currently called while resolving image-table entries. Issue #7923 reproduces a direct sampled-texture path (agc_driver_image_plane_compare_on_line): the Texture constructor creates a 1D D16 image with TRANSFER_DST | SAMPLED, which RADV reports unsupported via vkGetPhysicalDeviceImageFormatProperties; llvmpipe supports it. That direct path still reaches vkCreateImage without this preflight. I have a local candidate that queries support immediately before creation, but it overlaps this constructor hunk, so I’m holding it for sequencing. I have not run the RADV validation case locally.

stale+rebase

1. Freshness Envelope

• Head SHA: 1be24da14e8a0e9afecfa4f558018ce19d1bd895
• Merge-base SHA: 84c1a4d9a26429cb483ec963195cff8ef25d128b
• Behind origin/main: 11,944 commits
• Merge-tree status: exit code 1 (conflicts against origin/main in core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp, core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp, core/libs/prx/libSceAgcDriver/tests/Graphics.cpp, core/libs/prx/libSceAgcDriver/tests/GraphicsTests.hpp, and docs/dev/debt/libSceAgcDriver.md)

2. Findings

1. [verified] Merge conflicts require rebase: git merge-tree --write-tree origin/main 1be24da14e8a0e9afecfa4f558018ce19d1bd895 exits with code 1 due to conflicts in ShaderResources.cpp, Texture.cpp, Graphics.cpp, GraphicsTests.hpp, and libSceAgcDriver.md.
2. [verified] Conventions clean against merge-base: python tools/check_conventions.py --base 84c1a4d9a26429cb483ec963195cff8ef25d128b in an isolated scratch worktree reported 0 findings.
3. [read-only] Full template checklist compliance: PR description follows .github/pull_request_template.md with all checklist lines populated.
4. [read-only] Missing Vulkan validation layer comparison: AGENTS.md requires AGC PRs to report test runs with ANYPS5_REQUIRE_VULKAN=1, ANYPS5_NO_SHADER_CACHE=1, and VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation, confirming no new VUID- validation errors are introduced.
5. [author-reported] Reported extensive test additions in ImageTableEntryValidation.cpp covering memory preflight, allocation recovery, and faulting behavior on Windows Server 2025 / RTX 3060.

3. Cross-PR Ledger

• git merge-tree with open PR perf(libc): start the guest allocation overlap queries at the range that touches the address #6321 exits with code 0 (clean).
• Conflicts with open PRs perf(agc): reuse image-table elements whose descriptors repeat #6223, perf(agc): widen small epoch collects to their 64 KiB block on Windows #6273, fix(shader): read quad_perm DPP lanes in pixel shaders through quad operations #6290, and feat(agc): import cleared native multisample D32S8 targets and whole HTILE fills #6300 in shared driver execution files (ShaderResources.cpp, Texture.cpp, docs/dev/debt/libSceAgcDriver.md).

@fdfawef324 a title now needs this check on current main. God of War Ragnarök (PPSA08329, Windows, RTX 4080 SUPER) aborts with compute shader 0x1052a03100: std::bad_alloc right after [gpu] sampled view 0x4140000000 (4978x14883 format 28, dimension 4, slices 0..568) covers 161263 MiB.

• The entry is not a real texture. The image-table walk on an address-column table reads all 65,536 records from a base that moves with each dispatch. The real entries end at keys 4-17, and records further on are unrelated memory: dword 0 here is 0x41400000 (12.0f), and a later run decoded the same base as 12818x15412x368. cachedTexture copies the surface to host before any size check, so the 161 GiB view throws bad_alloc. The image-table path catches only std::runtime_error, so the dispatch dies even though the shader never selects that key.
• A narrow version of this PR's heap check fixes it: reject a view of 1 GiB or more that exceeds the device's largest heap before anything is read, so the entry faults only if it is selected. With that, a 2,700 s warm GWR run passed the old abort point (the same view appeared 10 times) with no image-table fault and kept presenting.

This PR has conflicts and has been 11,944 commits behind since 10-30. Do you plan to rebase it? If not, I can open the narrow check on its own, with a test that fails on main with std::bad_alloc, and credit this PR. MarcZX10's direct sampled-texture path (#7923) is not covered by the narrow version.

AI-assisted: yes (Claude Code)